1. Field of the Invention
The present invention relates to a line of sight detection device for detecting a line of sight of a user based on personal data related to an eyeball of the user, and to an apparatus having the line of sight detection device.
2. Related Background Art
Conventionally, there have been provided several types of devices (apparatuses) for detecting which position on a plane of observation a user or photographer looks at, i.e., for detecting a line of sight (visual axis) of the photographer, such as an eye camera. For example, in U.S. patent application No. 08/331,154, filed on Oct. 28, 1994, an observation point on which a photographer's gaze is fixed is obtained based on a cornea reflection image and an image focusing position of a pupil both detected by projecting parallel rays of light from a light source onto a front portion of the eyeball of the photographer. The same application also discloses an example of a single-lens reflex camera with an observation point detection device in which an automatic focus control of a photo-taking lens is performed by using information indicative of an observation point of the photographer.
FIG. 19 is a diagram explaining how a line of sight of a photographer is detected.
In FIG. 19, there are shown an eyeball 15, a cornea 16 and an iris 17 of the photographer. Referring now to the drawing, a method for detecting a line of sight of the photographer will be described.
At first, a beam of infrared light emitted from a light source 13b is applied to the cornea 16 of the photographer's eyeball 15 so that a cornea reflection image (virtual image) d can be formed by a portion of the infrared light reflected on the cornea 16. The cornea reflection image d is then converged on a light-receiving lens 12 and the converged image is focused at a position d' on an image sensor 14. Similarly, a beam of infrared light emitted from a light source 13a is applied to the cornea 16 of the eyeball 15 so that a cornea reflection image e can be formed by a portion of the infrared light reflected on the cornea 16. The cornea reflection image e is then converged on the light-receiving lens 12 and the converged image is focused at position e' on the image sensor 14.
On the other hand, beams from the ends a, b of the iris 17 are converged through the light-receiving lens 12 on the image sensor 14 so that images of the beams can be focused at positions a', b' on the image sensor 14, respectively. When the eyeball 15 has an optical axis of a small rotation angle .theta. to be rotated with respect to the optical axis of the light-receiving lens 12, if x-coordinates of the ends a, b of the iris 17 are xa, xb, respectively, the center point c of the pupil 19 has an x-coordinate xc represented as: EQU xc=(xa+xb)/2.
Further, an x-coordinate of a midpoint between the cornea reflection images d, e substantially corresponds to an x-coordinate xo of the center of cornea curvature. For this reason, if x-coordinates of the positions d, e at which the cornea reflection images are generated are xd, xe, respectively, and a standard distance between the center o of the cornea curvature and the center c of the pupil 19 is OC, a rotation angle .theta.x of the optical axis 15a of the eyeball 15 can meet the following relationship: EQU OC*sin .theta.x=(xd+xe)/2-xc. (1)
Thus, the rotation angle .theta. of the optical axis 15a of the eyeball 15 can be obtained by detecting the positions of respective characteristics of the eyeball 15 (the cornea reflection images and the center of the pupil) projected on the image sensor 14.
In other words, the rotation angle of the optical axis 15a of the eyeball 15 is obtained from the equation (1) as follows: EQU .beta.*OC*sin .theta.x{(x.sub.po -.delta.x)-x.sub.ic }*pitch,(2)
and, EQU .beta.*OC*sin .theta.y{(y.sub.po -.delta.y)-y.sub.ic }*pitch,(3)
where .theta.x is a rotation angle to rotate the optical axis of the eyeball on a z-x plane, .theta.y is a rotation angle to rotate the optical axis of the eyeball on a y-z plane, (x.sub.po, y.sub.po) are coordinates of the midpoint between the two cornea reflection images formed on the image sensor 14, (x.sub.ic, y.sub.ic) are coordinates of the center of the pupil taken on the image sensor 14, and pitch is a pitch between pixels on the image sensor 14. Further, .beta. is a focusing magnification which is determined based on the relative position of the eyeball 15 to the light-receiving lens 12 and factually obtained as a function of a gap between the two cornea reflection images.
.delta.x, .delta.y are correction terms for correcting an error of the coordinates of the midpoint between the cornea reflection images, the error produced when the eyeball of the photographer is illuminated with divergent rays instead of parallel rays. .delta.y also includes a correction term for correcting offset components produced by illuminating the eyeball of the photographer with the divergent rays from the lower eyelid side.
Next, an observation point on which the photographer's gaze is fixed is obtained from a relationship between the rotation angle of the eyeball and coordinates of the observation point. When the rotation angle of the optical axis of the photographer's eyeball is represented in coordinates (.theta.x, .theta.y), if the attitude of the camera is in a lateral posture, the photographer's observation point (x, y) on a plane of observation is obtained as follows: EQU x=m*(.theta.x+.DELTA.), (4)
and, EQU y=m*.theta.y, (5)
where an x direction and a y direction denote a horizontal direction and a vertical direction for the photographer, respectively, when the camera takes the lateral posture. Also, m is a coefficient of transformation for transforming the rotation angle of the eyeball into coordinates on a focusing plate, and .DELTA. is an angle between the optical axis 15a of the eyeball 15 and the visual axis (observation point). Since there are differences between individuals in the coefficients of transformation m and .DELTA., personal difference correction data for correcting personal differences needs to be predetermined in the form of data inclusive of these coefficients of transformation.
The personal difference correction data can be calculated for each photographer by detecting a rotation angle of the optical axis of the eyeball when the photographer looks at a given plane of coordinates. The detecting operation for obtaining the personal difference correction data is called calibration, and the personal difference correction data obtained by the calibration is called calibration data.
When a photographer looks at points on a plane of observation, the points having coordinates (x1, 0) and (x2, 0), if the rotation angles of the photographer's eyeball are represented in coordinates (.theta.x1, 0) and (.theta.x2, 0), respectively, the personal difference data can be obtained from the following equations: EQU m=(x1-x2)/(.theta.x1-.theta.x2), (6)
and, EQU .DELTA.=(x2.multidot..theta.x1-x1.multidot..theta.x2)/(x1-x2).(7)
The input of the personal difference data is performed in all postures of the camera such as in a posture in which the line of sight detection is not required. For example, since no photographer takes a picture with the camera turned upside down, the personal difference correction data cannot be input in a condition that the camera is turned upside down. Further, when the camera has such a hardware structure as the line of sight detection is prohibited when the camera takes a vertical posture, the personal difference data cannot be input in the vertical posture. Up to this time, various approaches to calibration have been proposed, such as a calibration process of keeping vertical and lateral calibration data, as disclosed in U.S. patent application No. 08/408,281, filed on Mar. 22, 1995. As to the relationship between the vertical and lateral postures, various proposals have been also made, such as a mechanism for prohibiting line of sight detection in the vertical posture and turning off an indication lamp, as disclosed in U.S. patent application No. 08/541,722, filed on Oct. 10, 1995, and a mechanism for changing lighting after detecting the attitude, as disclosed in U.S. patent application No. 08/386,200, filed on Feb. 9, 1995.
Such a line of sight detection device usable in vertical postures as well as in the lateral posture, however, cannot be expected to obtain an adequate accuracy in the vertical postures because the device uses the coefficients of personal differences related to the lateral posture. In the case that the line of sight detection device is used in the lateral posture, horizontal and vertical directions for the device correspond to those for the photographer, respectively. For this reason, the above conventional examples do not divide the coordinate system into a man coordinate system and a camera coordinate system. To clarify the discrimination between both coordinate systems, the man coordinate system is represented by adding e. FIG. 17 is a perspective view showing the man and camera coordinate systems when the camera is held in the lateral posture.
In the man coordinate system, the rotation angle of the eyeball is (.theta.x.sub.e, .theta.y.sub.e). The rotation angle is transformed into the camera coordinate system as follows:
for the lateral posture, (.theta.x.sub.e, .theta.y.sub.e).fwdarw.(.theta.x, .theta.y) (see FIGS. 18A1 and 18A2), PA1 for a vertical posture with the right side of the camera turned up, (.theta.x.sub.e, .theta.y.sub.e).fwdarw.(-.theta.y, +.theta.x) (see FIGS. 18B1 and 18B2), PA1 for a vertical posture with the right side of the camera turned down, (.theta.x.sub.e, .theta.y.sub.e).fwdarw.(+.theta.y, -.theta.x) (see FIGS. 18C1 and 18C2).
and,
When using the line of sight detection device as a standard for the plane of coordinates on which the photographer's gaze is fixed, a vertical direction for the photographer corresponds to the lateral direction for the line of sight detection device. Further, when taking into account that the coefficients of correction m and .DELTA. are determined for each eyeball of persons, the above equations (4) and (5) can be represented as: EQU x=m.sub.x .multidot.(.theta.x+.DELTA..sub.x), (4')
and, EQU y=m.sub.y .multidot.(.theta.y+.DELTA..sub.y). (5')
Since the values of m.sub.x, .DELTA..sub.x, m.sub.y and .DELTA..sub.y vary with the attitude of the camera, the calibration needs to be performed in each attitude of the camera for obtaining the personal difference correction data. Alternatively, other coefficients of transformation obtained in a corresponding attitude of another camera can be used for calibration in the attitude of the camera in which a line of sight of the photographer should be detected. As shown in the conventional examples, when the calibration is performed by assigning the observation point only to the x direction in the camera coordinate system, although the calculation is easier, strictly speaking, such calibration results in obtaining only the values of m.sub.x, .DELTA..sub.x in the above equation (4'). When using such a calibrating method, it is required to perform calibration in the vertical posture so that the values of m.sub.y, .DELTA..sub.y in the above equation (5') can be obtained. In other words, all the coefficients of transformation cannot be calculated unless the calibration in both postures is performed.